chore: Try to optimize compilation memory#10113
Conversation
0345079 to
3bd3138
Compare
There was a problem hiding this comment.
ACVM Benchmarks
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
purely_sequential_opcodes |
267650 ns/iter (± 1320) |
252791 ns/iter (± 1014) |
1.06 |
perfectly_parallel_opcodes |
232700 ns/iter (± 4288) |
224580 ns/iter (± 683) |
1.04 |
perfectly_parallel_batch_inversion_opcodes |
2784739 ns/iter (± 7785) |
2258668 ns/iter (± 4124) |
1.23 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Test Suite Duration
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
test_report_AztecProtocol_aztec-packages_noir-projects_aztec-nr |
119 s |
119 s |
1 |
test_report_AztecProtocol_aztec-packages_noir-projects_noir-contracts |
135 s |
135 s |
1 |
test_report_AztecProtocol_aztec-packages_noir-projects_noir-protocol-circuits_crates_blob |
261 s |
268 s |
0.97 |
test_report_AztecProtocol_aztec-packages_noir-projects_noir-protocol-circuits_crates_private-kernel-lib |
207 s |
212 s |
0.98 |
test_report_AztecProtocol_aztec-packages_noir-projects_noir-protocol-circuits_crates_reset-kernel-lib |
35 s |
35 s |
1 |
test_report_AztecProtocol_aztec-packages_noir-projects_noir-protocol-circuits_crates_types |
155 s |
132 s |
1.17 |
test_report_noir-lang_noir-bignum_ |
162 s |
162 s |
1 |
test_report_noir-lang_noir_bigcurve_ |
342 s |
421 s |
0.81 |
test_report_noir-lang_sha256_ |
15 s |
16 s |
0.94 |
test_report_noir-lang_sha512_ |
14 s |
13 s |
1.08 |
test_report_zkpassport_noir-ecdsa_ |
3 s |
1 s |
3 |
test_report_zkpassport_noir_rsa_ |
1 s |
1 s |
1 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Execution Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 909e22b | Previous: da35eca | Ratio |
|---|---|---|---|
rollup-block-root-single-tx |
0.003 s |
0.002 s |
1.50 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @TomAFrench
There was a problem hiding this comment.
Execution Time
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
private-kernel-inner |
0.012 s |
0.012 s |
1 |
private-kernel-reset |
0.157 s |
0.157 s |
1 |
private-kernel-tail |
0.01 s |
0.01 s |
1 |
rollup-block-root-first-empty-tx |
0.003 s |
0.003 s |
1 |
rollup-block-root-single-tx |
0.003 s |
0.003 s |
1 |
rollup-block-root |
0.004 s |
0.004 s |
1 |
rollup-checkpoint-merge |
0.003 s |
0.003 s |
1 |
rollup-checkpoint-root-single-block |
12.3 s |
12.8 s |
0.96 |
rollup-checkpoint-root |
12.5 s |
12.4 s |
1.01 |
rollup-root |
0.004 s |
0.004 s |
1 |
rollup-tx-base-private |
0.299 s |
0.299 s |
1 |
rollup-tx-base-public |
0.236 s |
0.238 s |
0.99 |
rollup-tx-merge |
0.002 s |
0.002 s |
1 |
semaphore-depth-10 |
0.009 s |
0.008 s |
1.13 |
sha512-100-bytes |
0.069 s |
0.071 s |
0.97 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Opcode count
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
private-kernel-inner |
15876 opcodes |
15876 opcodes |
1 |
private-kernel-reset |
78291 opcodes |
78291 opcodes |
1 |
private-kernel-tail |
12195 opcodes |
12195 opcodes |
1 |
rollup-block-root-first-empty-tx |
1365 opcodes |
1365 opcodes |
1 |
rollup-block-root-single-tx |
1049 opcodes |
1049 opcodes |
1 |
rollup-block-root |
2410 opcodes |
2410 opcodes |
1 |
rollup-checkpoint-merge |
2330 opcodes |
2330 opcodes |
1 |
rollup-checkpoint-root-single-block |
962022 opcodes |
962022 opcodes |
1 |
rollup-checkpoint-root |
963382 opcodes |
963382 opcodes |
1 |
rollup-root |
2830 opcodes |
2830 opcodes |
1 |
rollup-tx-base-private |
263910 opcodes |
263910 opcodes |
1 |
rollup-tx-base-public |
245349 opcodes |
245349 opcodes |
1 |
rollup-tx-merge |
1486 opcodes |
1486 opcodes |
1 |
semaphore-depth-10 |
5700 opcodes |
5700 opcodes |
1 |
sha512-100-bytes |
13173 opcodes |
13173 opcodes |
1 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Artifact Size
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
private-kernel-inner |
736.2 KB |
736.2 KB |
1 |
private-kernel-reset |
2111.5 KB |
2111.5 KB |
1 |
private-kernel-tail |
549.6 KB |
549.6 KB |
1 |
rollup-block-root-first-empty-tx |
179.9 KB |
179.9 KB |
1 |
rollup-block-root-single-tx |
178.2 KB |
178.2 KB |
1 |
rollup-block-root |
258.2 KB |
258.2 KB |
1 |
rollup-checkpoint-merge |
373.8 KB |
373.8 KB |
1 |
rollup-checkpoint-root-single-block |
27645.4 KB |
27645.4 KB |
1 |
rollup-checkpoint-root |
27690.6 KB |
27690.6 KB |
1 |
rollup-root |
414.3 KB |
414.3 KB |
1 |
rollup-tx-base-private |
4920.2 KB |
4920.2 KB |
1 |
rollup-tx-base-public |
4571.5 KB |
4571.5 KB |
1 |
rollup-tx-merge |
186.2 KB |
186.2 KB |
1 |
semaphore-depth-10 |
570.7 KB |
570.7 KB |
1 |
sha512-100-bytes |
506.3 KB |
506.3 KB |
1 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Compilation Time
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
private-kernel-inner |
1.68 s |
1.722 s |
0.98 |
private-kernel-reset |
7.846 s |
7.59 s |
1.03 |
private-kernel-tail |
1.338 s |
1.538 s |
0.87 |
rollup-block-root-first-empty-tx |
1.422 s |
1.382 s |
1.03 |
rollup-block-root-single-tx |
1.47 s |
1.38 s |
1.07 |
rollup-block-root |
1.49 s |
1.46 s |
1.02 |
rollup-checkpoint-merge |
1.526 s |
1.478 s |
1.03 |
rollup-checkpoint-root-single-block |
223 s |
210 s |
1.06 |
rollup-checkpoint-root |
219 s |
213 s |
1.03 |
rollup-root |
1.514 s |
1.528 s |
0.99 |
rollup-tx-base-private |
17.54 s |
17.92 s |
0.98 |
rollup-tx-base-public |
131.4 s |
80.42 s |
1.63 |
rollup-tx-merge |
1.348 s |
1.384 s |
0.97 |
semaphore-depth-10 |
0.799 s |
0.786 s |
1.02 |
sha512-100-bytes |
1.587 s |
2.05 s |
0.77 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Compilation Memory
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
private-kernel-inner |
266.47 MB |
273.53 MB |
0.97 |
private-kernel-reset |
498.21 MB |
562.99 MB |
0.88 |
private-kernel-tail |
248.46 MB |
256.79 MB |
0.97 |
rollup-block-root-first-empty-tx |
341.01 MB |
343.28 MB |
0.99 |
rollup-block-root-single-tx |
338.46 MB |
340.51 MB |
0.99 |
rollup-block-root |
341.44 MB |
344.33 MB |
0.99 |
rollup-checkpoint-merge |
340.78 MB |
344.37 MB |
0.99 |
rollup-checkpoint-root-single-block |
6820 MB |
9710 MB |
0.70 |
rollup-checkpoint-root |
6820 MB |
9720 MB |
0.70 |
rollup-root |
345.05 MB |
346.97 MB |
0.99 |
rollup-tx-base-private |
1050 MB |
1360 MB |
0.77 |
rollup-tx-base-public |
2890 MB |
6960 MB |
0.42 |
rollup-tx-merge |
338.65 MB |
339.37 MB |
1.00 |
semaphore_depth_10 |
92.18 MB |
97.55 MB |
0.94 |
sha512_100_bytes |
185.49 MB |
251.64 MB |
0.74 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
Execution Memory
Details
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
private-kernel-inner |
255.57 MB |
255.57 MB |
1 |
private-kernel-reset |
293.11 MB |
293.11 MB |
1 |
private-kernel-tail |
244 MB |
244 MB |
1 |
rollup-block-root |
338.92 MB |
338.92 MB |
1 |
rollup-checkpoint-merge |
337.79 MB |
337.79 MB |
1 |
rollup-checkpoint-root-single-block |
1020 MB |
1020 MB |
1 |
rollup-checkpoint-root |
1020 MB |
1020 MB |
1 |
rollup-root |
338.99 MB |
338.99 MB |
1 |
rollup-tx-base-private |
452.47 MB |
452.47 MB |
1 |
rollup-tx-base-public |
467.36 MB |
467.36 MB |
1 |
rollup-tx-merge |
337.18 MB |
337.18 MB |
1 |
semaphore_depth_10 |
73.69 MB |
73.69 MB |
1 |
sha512_100_bytes |
71.96 MB |
71.96 MB |
1 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Compilation Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
rollup-tx-base-public |
131.4 s |
80.42 s |
1.63 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @TomAFrench
jfecher
left a comment
There was a problem hiding this comment.
Looks great! Showing some good improvements in memory usage in CI.
… better explanation
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Test Suite Duration'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
test_report_zkpassport_noir-ecdsa_ |
3 s |
1 s |
3 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @TomAFrench
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'ACVM Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: 0c5c7da | Previous: 845a8e4 | Ratio |
|---|---|---|---|
perfectly_parallel_batch_inversion_opcodes |
2784739 ns/iter (± 7785) |
2258668 ns/iter (± 4124) |
1.23 |
This comment was automatically generated by workflow using github-action-benchmark.
CC: @TomAFrench
Automated pull of nightly from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE chore: add unit tests for brillig-gen (noir-lang/noir#10130) chore(ACIR): remove non-ACIR intrinsics during simplification (noir-lang/noir#10145) fix: Do not carry over `#[fold]` to unconstrained functions during monomorphization (noir-lang/noir#10155) chore(SSA): avoid consuming self when returning Arc (noir-lang/noir#10147) fix(ssa): Use `Type::element_size` instead of `Type::flattened_size` for `optimize_length_one_array_read` (noir-lang/noir#10146) fix(ssa): SSA Interpreter handle overflow by promoting to Field (noir-lang/noir#10097) chore: Try to optimize compilation memory (noir-lang/noir#10113) chore(ACIRgen): smaller AcirDynamicArray value_types (noir-lang/noir#10128) chore(brillig_vm): Re-org integration tests and add a couple more (noir-lang/noir#10129) chore: unhide `inliner-aggressiveness` option (noir-lang/noir#10137) chore(brillig_vm): Expand arithmetic int ops tests and add field ops tests (noir-lang/noir#10101) chore(ACIR): don't override output count in black box function (noir-lang/noir#10123) chore(test): Add `interpret_execution_failure` tests (noir-lang/noir#9912) fix(ACIR): correctly display the zero expression (noir-lang/noir#10124) chore: typos and some refactors in `acvm/src/pwg/mod.rs` (noir-lang/noir#10055) chore: add brillig_call submodule (noir-lang/noir#10108) chore(ACIRgen): always compute array offset (noir-lang/noir#10099) chore: More BTreeSet avoidance (noir-lang/noir#10107) END_COMMIT_OVERRIDE Co-authored-by: Tom French <15848336+TomAFrench@users.noreply.github.com>
Automated pull of nightly from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE chore: add unit tests for brillig-gen (noir-lang/noir#10130) chore(ACIR): remove non-ACIR intrinsics during simplification (noir-lang/noir#10145) fix: Do not carry over `#[fold]` to unconstrained functions during monomorphization (noir-lang/noir#10155) chore(SSA): avoid consuming self when returning Arc (noir-lang/noir#10147) fix(ssa): Use `Type::element_size` instead of `Type::flattened_size` for `optimize_length_one_array_read` (noir-lang/noir#10146) fix(ssa): SSA Interpreter handle overflow by promoting to Field (noir-lang/noir#10097) chore: Try to optimize compilation memory (noir-lang/noir#10113) chore(ACIRgen): smaller AcirDynamicArray value_types (noir-lang/noir#10128) chore(brillig_vm): Re-org integration tests and add a couple more (noir-lang/noir#10129) chore: unhide `inliner-aggressiveness` option (noir-lang/noir#10137) chore(brillig_vm): Expand arithmetic int ops tests and add field ops tests (noir-lang/noir#10101) chore(ACIR): don't override output count in black box function (noir-lang/noir#10123) chore(test): Add `interpret_execution_failure` tests (noir-lang/noir#9912) fix(ACIR): correctly display the zero expression (noir-lang/noir#10124) chore: typos and some refactors in `acvm/src/pwg/mod.rs` (noir-lang/noir#10055) chore: add brillig_call submodule (noir-lang/noir#10108) chore(ACIRgen): always compute array offset (noir-lang/noir#10099) chore: More BTreeSet avoidance (noir-lang/noir#10107) END_COMMIT_OVERRIDE
Automated pull of nightly from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE chore: add unit tests for brillig-gen (noir-lang/noir#10130) chore(ACIR): remove non-ACIR intrinsics during simplification (noir-lang/noir#10145) fix: Do not carry over `#[fold]` to unconstrained functions during monomorphization (noir-lang/noir#10155) chore(SSA): avoid consuming self when returning Arc (noir-lang/noir#10147) fix(ssa): Use `Type::element_size` instead of `Type::flattened_size` for `optimize_length_one_array_read` (noir-lang/noir#10146) fix(ssa): SSA Interpreter handle overflow by promoting to Field (noir-lang/noir#10097) chore: Try to optimize compilation memory (noir-lang/noir#10113) chore(ACIRgen): smaller AcirDynamicArray value_types (noir-lang/noir#10128) chore(brillig_vm): Re-org integration tests and add a couple more (noir-lang/noir#10129) chore: unhide `inliner-aggressiveness` option (noir-lang/noir#10137) chore(brillig_vm): Expand arithmetic int ops tests and add field ops tests (noir-lang/noir#10101) chore(ACIR): don't override output count in black box function (noir-lang/noir#10123) chore(test): Add `interpret_execution_failure` tests (noir-lang/noir#9912) fix(ACIR): correctly display the zero expression (noir-lang/noir#10124) chore: typos and some refactors in `acvm/src/pwg/mod.rs` (noir-lang/noir#10055) chore: add brillig_call submodule (noir-lang/noir#10108) chore(ACIRgen): always compute array offset (noir-lang/noir#10099) chore: More BTreeSet avoidance (noir-lang/noir#10107) END_COMMIT_OVERRIDE
Automated pull of nightly from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE chore: add unit tests for brillig-gen (noir-lang/noir#10130) chore(ACIR): remove non-ACIR intrinsics during simplification (noir-lang/noir#10145) fix: Do not carry over `#[fold]` to unconstrained functions during monomorphization (noir-lang/noir#10155) chore(SSA): avoid consuming self when returning Arc (noir-lang/noir#10147) fix(ssa): Use `Type::element_size` instead of `Type::flattened_size` for `optimize_length_one_array_read` (noir-lang/noir#10146) fix(ssa): SSA Interpreter handle overflow by promoting to Field (noir-lang/noir#10097) chore: Try to optimize compilation memory (noir-lang/noir#10113) chore(ACIRgen): smaller AcirDynamicArray value_types (noir-lang/noir#10128) chore(brillig_vm): Re-org integration tests and add a couple more (noir-lang/noir#10129) chore: unhide `inliner-aggressiveness` option (noir-lang/noir#10137) chore(brillig_vm): Expand arithmetic int ops tests and add field ops tests (noir-lang/noir#10101) chore(ACIR): don't override output count in black box function (noir-lang/noir#10123) chore(test): Add `interpret_execution_failure` tests (noir-lang/noir#9912) fix(ACIR): correctly display the zero expression (noir-lang/noir#10124) chore: typos and some refactors in `acvm/src/pwg/mod.rs` (noir-lang/noir#10055) chore: add brillig_call submodule (noir-lang/noir#10108) chore(ACIRgen): always compute array offset (noir-lang/noir#10099) chore: More BTreeSet avoidance (noir-lang/noir#10107) END_COMMIT_OVERRIDE
Description
Problem*
The
rollup-checkpoint-rootandrollup-checkpoint-root-single-blockcontracts need almost 10GB memory during compilation, see here.Summary*
Improves compilation memory usage in the heaviest aztec-protocol circuits:
Reinsertion
Adds an
allow_reinsertflag toFunctionInserter::push_instruction_value; when true it passes the instruction ID to toDataFlowGraph::insert_instruction_and_results_if_simplifiedto try to avoid storing a new copy of an instruction when we can reinsert it as-is.The flag gets different values in different passes:
flatten_cfg: Allows reinsertion as we are mostly focussing on sprinkling around theenable_side_effectinstructions and weaving them into instructions, but whatever isn't affected can be reused. Not using OTOH duplicates all instructions.mem2reg: We can only reuse instructions we haven't seen before, because we are tracking which instruction to remove by ID, and if we reinsert multiple times then we might just want to remove the last one, yet we end up removing all occurrences.unrolling: Not using because we are importing between blocksloop_invariant: Using when we reinsert into the same block, not when hoisting into the pre-header.Inline order
Changed function inlining to go in bottom-up order and overwrite functions as soon as they have been inlined, hoping to reduce the outstanding memory. Unfortunately it had no effect on the peak memory.
Remove unused instructions
Added a
Ssa::remove_unused_instructionspass after some of the DIE and mem2reg passes, hoping to free some memory from theDenseMapby making the content smaller. It had negligible effect, maybe 300MB. Potentially theInstruction::Noopis as large as the largestInstructionanyway, except forMakeArraywhich has aVecin it, and maybe some of the others that containString; the results wereSmallVec, so they don't consume more or less if there are just 1-2 results.Instruction::map_valuesBased on profiling, the unconditional creation of a new
im::VecforMakeArray::elementsconsumed a lot of memory. Sinceimsupports sharing viaArc, I changed this to try to just.clone()and not modify unless an item actually changed.Additional Context
Profiled command:
Code to compile and sign
nargofor Instruments:Documentation*
Check one:
PR Checklist*
cargo fmton default settings.